Synchronous parallel pixel processing for scalable color reproduction systems

ABSTRACT

What is disclosed is a novel system and method for parallel processing of intra-image data in a distributed computing environment. A generic architecture and method are presented which collectively facilitate image segmentation and block sorting and merging operations with a certain level of synchronization in a parallel image processing environment which has been traditionally difficult to parallelize. The present system and method enables pixel-level processing at higher speeds thus making it a viable service for a print/copy job document reproduction environment. The teachings hereof have been simulated on a cloud-based computing environment with a demonstrable increase of ≈2× with nominal 8-way parallelism, and an increase of ≈20×-100× on a graphics processor. In addition to production and office scenarios where intra-image processing are likely to be performed, these teachings are applicable to other domains where high-speed video and audio processing are desirable.

TECHNICAL FIELD

The present invention is directed to systems and methods which processintra-image pixel-level data on a parallel image processing architecturesuch as multi-processor, multi-core, or multi-threaded systems.

BACKGROUND

Conventional color controls architectures and algorithms do not offerpixel level processing with the advantages of synchronization amongstworkers operating on pieces of an image and automated techniques forscalable parallelization. Image processing, especially color controls,on pixel ensembles of the order of 10⁸-10⁹ per image are highlydata-intensive and computation-intensive requiring scalableparallelization techniques. Some algorithms are not designed to operatein parallel, especially offering advantages like synchronization. Thetype of parallelization required to address these problems is calledconnected parallel (involving inter-processor communication) in additionto data parallel stages. Automatically dealing with connected parallelcomputation is generally hard in parallelization and have to bespecifically addressed for the data sizes we consider in color control.Approximation techniques such as interpolation are used in colorcontrols, some of them to deal with the large image sizes. The use ofsuch techniques may result in artifacts and adversely affect in imagequality. However, it is desirable to avoid interpolation by processingeach pixel individually. Due to large image sizes, failure in a certainstage of processing is not uncommon. A number of such problems at thestage level is because of the asynchronous nature of the processing andcould benefit from a small number of retries.

What is needed in this art are increasingly sophisticated systems andmethods which facilitates parallel processing of intra-image data in adistributed computing environment.

INCORPORATED REFERENCES

The following U.S. Patents, U.S. Patent Applications, and Publicationsare incorporated herein in their entirety by reference.

“k-means++: The Advantages of Careful Seeding”, David Arthur and SergeiVassilvitskii, ACM-SIAM Symposium On Discrete Algorithms (SODA), NewOrleans, La., (Jan. 7-9, 2007).

BRIEF SUMMARY

What is disclosed is a novel system and method for parallel processingof intra-image data in a distributed computing environment. A genericarchitecture and method are presented which collectively facilitateimage segmentation and block sorting and merging operations in aparallel image processing environment which has been traditionallydifficult to parallelize. Broadly defined, the present method involvesreceiving an image to be processed and segmenting the image into aplurality of image segments. The image segments are then processed inparallel into a plurality of batches of processed pixels. The processedpixels are sorted into sorted batches which, in turn, are merged into aplurality of merged batches. The merged batches are concatenated toproduce a processed output image. The present system and method enablespixel-level processing at higher speeds thus making it a viable servicefor a print/copy job document reproduction environment. The teachingshereof have been simulated on a cloud-based computing environment with ademonstrable increase of ˜2× with nominal 8-way parallelism, and anincrease of ˜20×-100× on a general purpose graphics processing unit(GPU). In addition to production and office scenarios where intra-imageprocessing are likely to be performed, these teachings are applicable toother domains where high-speed video and audio processing are desirable.

In one example embodiment, the present method for parallel processing ofintra-image data in a distributed computing environment involves thefollowing. Customer expectation criteria are received. Customerexpectation criteria may comprise, for example, a desired or targetprice for their print/copy job, a desired or target timeframeexpectation in which their job is to be processed (i.e., response time),a particular type of processing to be performed on their job, and/or adesired quality of the processed output image. Such a system uses thestate of the customer job and the state of the process and/or printingdevice to create the output. This is called state-based imaging system.A set of resources d in a state-based imaging system having w availableresources are determined via an optimization process which meet thecustomer criteria. The selection of the desired number of availableresources may be based upon a prior run history of similar jobs. Thestate-based imaging system has a plurality of segmentation resources forsegmentation processing of an input image, a multiple of k resourcesavailable for pixel-level processing an output of each of thesegmentation resources, a plurality of sorting resources for sorting anoutput of each of the pixel-level processors, and a plurality of mergingresources capable of hierarchically merging the output of each of thesorting resources to produce a processed image output. A customerbilling strategy is implemented based upon the set of resources used toprocess the customer job {R}, the number of pixels in the received imageP, the total number of pixels processed by each of the pixel-levelprocessing resources P_(i), where i is the index for the pixelprocessing resources, the total number of sorted batches produced by thesorting resources B_(s), the total number of batches merged by themerging resources B_(m), and/or a size of the processed output image O,i.e., Price=function({R}, P, P_(i), B_(s), B_(m), O). Variousembodiments are disclosed.

Many features and advantages of the above-described method will becomereadily apparent from the following detailed description andaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other features and advantages of the subject matterdisclosed herein will be made apparent from the following detaileddescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 illustrates a high level block diagram of an example state-basedimaging system showing three important parallelized stages;segmentation, pixel processing and merging;

FIG. 2 illustrates an example of scalable distributed state-basedimaging system comprising a plurality of parallelized resourcescomprising processors, threads, modules, and/or code segments;

FIG. 3 is a flow diagram of one example embodiment of the present methodfor parallel processing of intra-image data in a distributed computingenvironment;

FIG. 4 is a flow diagram of one example embodiment of a resourceavailability-aware distributed k-means segmentation for connectedparallel computation in the state-based imaging system;

FIG. 5 is a continuation of the flow diagram of FIG. 4 with processingcontinuing with respect to node A;

FIG. 6 depicts one example embodiment of a distributed (hierarchical)merge operation in accordance with the method described with respect toFIG. 5; and

FIG. 7 illustrates a block diagram of one example embodiment of aspecial purpose computer system useful for implementing one or moreaspects of the resources of FIGS. 1 and 2 as further described withrespect to the various flow diagrams of FIGS. 4 and 5.

DETAILED DESCRIPTION

What is disclosed is a novel system and method for parallel processingof intra-image data in a distributed computing environment. A genericarchitecture and method are presented which collectively facilitateimage segmentation and block sorting and merging operations in aparallel image processing environment which has been traditionallydifficult to parallelize.

It should be understood that one of ordinary skill in this art would bereadily familiar with many facets of color science and image processingand other techniques and algorithms commonly found in the color scienceand document reproduction arts. Those of ordinary skill would befamiliar with the text: “Digital Color Imaging Handbook”, 1st Ed., CRCPress (2003), ISBN-13: 97808-4930-9007, and “Control of Color ImagingSystems: Analysis and Design”, CRC Press (2009), ISBN-13:97808-4933-7468, both of which are incorporated herein in their entiretyby reference.

Non-Limiting Definitions

A “print/copy job” (or simply “job”) as used herein, refers to a logicalunit of work that is to be completed for a customer. A job may comprise,for example, an image, a document, a color brochure, photo, film, andthe like.

“Segmentation” is the process of partitioning a digital representationof a job into sets of pixels (also referred to as super-pixels) thathave a specific group property. The goal of segmentation is to simplifyor change the representation of a digitized job into something that ismore easily analyzed. The quality of the segmentation depends on thecomposition of the job being processed. For instance, smoothly shadedsurfaces of a digital image comprising clear gray-level steps betweendifferent surfaces are more readily segmentable.

“Image segmentation” is a technique which clusters pixels into salientimage regions, i.e., regions corresponding to individual surfaces,objects, or natural parts of objects. Image segmentation assigns a labelto every pixel in an image such that pixels with the same label sharecertain visual characteristics. The result of image segmentation is aset of segments that collectively cover the entire image, or a set ofcontours extracted from the image. Each of the pixels in a region aresimilar with respect to some characteristic or computed properly, suchas color, intensity, texture, and the like. Adjacent regions aresignificantly different with respect to the same characteristic(s). Insemi-automatic segmentation, a user outlines one or more regions ofinterest within the image and segmentation is applied so that the paththat best fits the edge of the image is shown to the user for furtherselection and/or confirmation.

“Clustering” is a commonly used procedure in data mining algorithms.Practical applications of clustering include unsupervised classificationand taxonomy generation, nearest neighbor searching, scientificdiscovery, vector quantization, text analysis, and navigation.

A “processor core” is a term used herein which is intended to encompassone processing entity of a multi-core, multi-processor, and/ormulti-threaded parallel processing environments.

“Concatenation” refers to any process in which two or more data sets,generated in accordance herewith, are combined, compared, or areotherwise used together in any way.

Brief General Discussion

As discussed in the background section hereof, high quality imagerendering systems try to take advantage of regions and groups of pixelsin images. Such systems use different techniques for rendering,especially using different profiles for different states or objects ascan be defined in the image. Parallelizing at the page level is a firststep. Beyond that, parallelizing at the image level introduces issueswith respect to communicating image level details to a plurality ofparallel processors. Such issues may take the form of non-trivialdependencies, and the like, which are difficult to manage. At the imagelevel, the problem becomes a mixture of connected parallel stagesinvolving inter-processor or inter-core communication & data parallelstages. While parallelization produces benefits, the image processingsystem needs to be modular as it may be important, at run-time, to beable to add additional processing resource. Moreover, the imageprocessing system needs to have a fair degree of fault tolerance, asresources may be deleted due to natural causes (resource unavailability)or unexpected causes (resource failure). Such capability to add/deleteresources for performance reasons without needing to change code istermed scalable and fault-tolerant parallelization. Scalability requiresextensive use of multi-core processors, multi-processors and/or generalpurpose graphics processing units (GPU). This is particularly true forimage processing where a given image may consume several gigabytes ofmemory or storage to manipulate and process, including swap space andmeta-data such as profiles, regions in the image, etc. Resortingentirely to a cloud-type computing environment can introduce bandwidthconcerns. For a single production quality image-heavy document at 600dpi, a typical 25 page document may consist of approximately 1.5×10¹⁰pixels (15 billion) or ˜392 GB. Automatic pixel or image levelparallelization is also required as each image may take ˜40 min toprocess (e.g. image at 600 dpi).

What is provided herein is a generic multi-stage state-based imageprocessing system which effectuates parallelized image segmentation,pixel-level processing, and sort/merge operations in an image processingenvironment. In a first stage, a process of image segmentation dividedthe job (digital image) into segments of substantially similar sizes;each containing meta-data about color clusters of the image. In a secondstage, the segments are independently color converted throughcluster-specific processing. In a third stage, the clustered segmentsare sorted and hierarchically merged back into an output image. Bymaking the number of segments in the first or second stages a multipleof the number of available resources (multi-core, multi-processor,and/or multi-threaded resources), and by utilizing a hierarchical mergeto reconstruct the image, load balancing and synchronization can beeffectively achieved. By automatic selection of the degree ofparallelization, synchronization is produced in connected parallel taskshaving inter-processor communication across iterations. The presentsystem can be implemented on existing platforms such as MPI (MessagePassing Interface).

Note that merely creating equal portions of work does not producesynchronization and may even degrade performance. Synchronization can beinterpreted as all of the processors that are working on a stage ascompleting more or less at the same time. Pixel-level processing is dataparallel and can be split into as many pieces as possible subject toprocessor and bandwidth availability. But synchronization will occuronly when the load is suitably allocated.

The outputs of the pixel-level processors are sorted so as to enableefficient merging in linear time. For a large number of sorted lists mto be merged, such time is given as O(p log m) where p is the totalnumber of pixels in a merged batch. Merging can occur hierarchicallybecause of the separation. Portions of the image are streamed batch bybatch, i.e., first p pixels are processed then the next p pixels, and soforth, such that the resulting image can be quickly and efficientlyreconstructed using, for instance, a process of simple concatenation.The present method allows for logical delineation of segments producingbenefits in pixel processing and merging.

Example State-Based Image Processing System

Reference is now being made to FIG. 1 which illustrates a high levelblock diagram of an example state-based image processing system. Theexample illustrated state-based system 100 is shown comprising threeprimary stages, i.e., a segmentation stage 102, a pixel-processing stage104, and a merging stage 106. Job segmentation module 102 receives animage (job) 108 and performs image segmentation on the received image.Image segments (or clusters) are passed to a plurality of pixel-levelprocessor, collectively at 104, wherein the color pixels are processedin a parallel manner. The results thereof are provided to a job segmentmerge 106 wherein the processed image segments are merged to produce acolor corrected output 110.

Example Distributed Computing Environment

Reference is now being made to FIG. 2 which illustrates one examplescalable distributed state-based imaging system 200 comprising aplurality of parallelized resources (or modules). The state-basedimaging system is a distributed computing environment and may be, forexample, a plurality of processors, a plurality of multi-coreprocessors, and/or a multi-threaded system. Any of the availableresources in the system may comprise one or more distributed codesegments.

Received image 202 is provided to a distributed segmentation system 204which performs image segmentation in a parallel manner. In variousembodiments, the number of segments into which the received image isdivided is automatically selected as a function of processoravailability. At the end of each iteration (of segmentation),information regarding a previous step is shared and updated. Preferably,the information shared comprises meta data which is less bandwidthintensive. The image segments are provided to a plurality of pixelprocessors 206 wherein processing of the color pixels is performed.Processing of the image segments can be performed with/without furthersub-division of the image segments. The results thereof are provided toa plurality of sorted batch code segments 208 which, in a manner morefully described herein, sort the processed pixels into batches of sortedpixels. Such a distributed sorter configuration can sort processedpixels into batches. The sorted processed pixels are provided, inordered batches, to a distributed, hierarchical merge resource 210 whichproduces merged batches which are concatenated to produce output image212. If we let n be the total number of pixels in an image (this is ofthe order of hundreds of millions) and we let p be the number of pixelsdesired in a single batch of output, the distributed sorter 208 can sortthe image into several batches such that output of the segment processedby each pixel processor 206 is in ordered batches comprisingapproximately (n div p)+1 batches, with each batch having at most ppixels.

It should be appreciated that any of the components (or ‘modules’) ofany of the embodiments of FIGS. 1 and 2 may designate a resourcecomprised of software or hardware, or a combination of both. A modulemay be implemented using a single software program comprising, at leastin part, machine readable program instruction, or a single piece ofhardware such as an ASIC, GPU, FPGA, electronic circuit, or specialpurpose processing system as shown in FIG. 7. A plurality of modules maycollectively perform a given function. A plurality of modules may beexecuted by either a single computer system or a plurality of computersystems operating in parallel. Connections between modules include bothphysical and logical connections.

Flow Diagram of First Embodiment

Reference is now being made to the flow diagram of FIG. 3 whichillustrates one example embodiment of the present method for parallelprocessing of intra-image data in a distributed (preferablyself-optimized and autonomic) computing environment. Flow processingstarts at 300 and immediately proceeds to step 302.

At step 302, an image to be processed is received in a state-basedimaging system having a set of w available resources. One example systemis shown and discussed with respect to FIG. 2. The image can be receivedfrom a remote device over a network such as a WIFI, LAN, or Internet.The image may alternatively be retrieved from memory or storage.

At step 304, operational parameters are computed which ensuresynchronization of the segmentation, pixel processing, and mergingoperations. Operational parameters are based on the parallel processingenvironment wherein the methods hereof find their intended uses. Oneskilled in this art would appreciate that operation parameters generatedfor a parallel processing environment can, at least in part, be based ona wide variety of factors such as, for example, image characteristics,resource availability, and quality of service required.

At step 306, the image is segmented into a plurality of image segmentsusing, for example, the distributed segmentation system 204 whereinimage segmentation is performed in a parallelizable manner.

At step 308, the image segments are processed into a plurality ofbatches of processed pixels using, for example, the plurality of pixelprocessors 206 wherein processing of the color pixels is performed in aparallelizable manner. Processing of the image segments can be performedwith/without further sub-division of the image segments. Pixel-levelprocessing is data parallel and can be split into as many pieces aspossible for processing subject to processor and bandwidth constraints.The processed pixels are provided as an output in the form of batches ofpixels. It should be understood that the term “processing” refers to anytransformation, whether mathematical or logical, and also is intended toencompass GPU operations including associated transfers.

At step 310, the batches of processed pixels are sorted into a pluralityof ordered batches using, for example, the plurality of sorted batchcode segments 208 which sort the processed pixels into batches of sortedpixels. Such a distributed sorter configuration can sort processedpixels into ordered batches.

At step 312, the ordered batches are hierarchically merged into aplurality of merged batches using, for example, the distributed,hierarchical merge resource 210 which produces merged batches.

At step 314, the merged batches are concatenated to produce a processedoutput image. Thereafter, in this embodiment, further processing stops.

Flow Diagram of Second Embodiment

Reference is next being made to the flow diagram of FIG. 4 whichillustrates one example embodiment of a resource availability-awaredistributed k-means segmentation for connected parallel computation inthe state-based imaging system having w available resources. Each of theavailable resources can process a code segment in parallel with theothers. Processing begins at step 400 and immediately proceeds to step402.

At step 402, receive a customer job and at least one customerexpectation criteria. The job comprises at least one image to beprocessed in a state-based imaging system having a set of w availableresources. One example system is shown and discussed with respect toFIG. 2. The image can be received from a remote device over a networksuch as a WIFI, LAN, or Internet. The image may alternatively beretrieved from memory or storage. The customer criteria may take theform of, for example, a desired or target price point for the customerjob, a desired or target timeframe in which the customer job is to beprocessed, a type of processing desired to be performed on the customerjob, and/or a desired quality of the processed output image.

At step 404, a desired number of resources d are selected from a set ofavailable resources w in a state-based image processing system whichmeet or exceed the customer's expectation criteria. A speed requirements may also be estimated for this job. Resource selection may bedetermined from run histories of similar jobs. The resources arepreferably selected via a dynamic optimization strategy. The state-basedimaging system has a plurality of image segmentation resources, amultiple of k resources available for pixel-level processing, aplurality of sorting resources, and a plurality of merging resources.

At step 406, the image is divided into k=min(w,d) image segments. Ifeach image segment is available as more sub-divided files, then therewill still be an integral multiple of k files to process. A k-meansimage segmentation is pre-employed such that k is selected in a mannerwhich apportions work for distributed segmentation 204 of FIG. 2 to keepthe parallel segmentation process in synchronization in a systemcomprising an integral multiple of k segments and k pixel-levelprocessors. The image can be stored as k parts in a distributed filesystem. Segments are preferably stored as separate files but this maynot be desirable in all cases. As a bandwidth optimization, segments maybe logically delineated (storing meta-data concerning segment for eachpixel) as opposed to being physically separated. As an example, segmentinformation may be stored along with each core of the GPU system asassociated metadata. This avoids transportation of files or segments forsubsequent stages. This optimization produces efficiencies in subsequentpixel-level processing, sorting, and merge stages. Preserving theoriginal order comes with a penalty i.e., the need to store another fileof the order of 2*n where n is of the order of 10⁹, decisions such asthis have to be weighed carefully.

At step 408, read each image segment and sort the points to each globalcentroid by a minimum squared distance. Each resource operates on eachpart of the image in parallel and goes through each pixel and associatesthe pixel to one of k known global centroids according to a nearest meansquared distance. The k-means algorithm is utilized because thisalgorithm tries to minimize the average squared distance between pointsin the same cluster. The k-means algorithm is a procedure for clusteringdata sets which assumes that the data objects to be clustered areavailable as points (or vectors) in a d-dimensional Euclidean space. Thek-means algorithm seeks a minimum variance grouping of data thatminimizes the sum of squared Euclidean distances from certain clustercentroids. The K-means algorithm is an iterative technique that is usedto partition an image into K clusters. In one embodiment, the basicalgorithm is as follows:

-   -   1) Pick K cluster centers (either randomly or based on some        heuristic).    -   2) Assign each pixel in the image to the cluster that minimizes        the variance between the pixel and the cluster center.    -   3) Re-compute the cluster centers by averaging all of the pixels        in the cluster.    -   4) Repeat steps 2 and 3 until convergence is attained (e.g. no        pixels change clusters).

In this case, variance is the squared or absolute difference between apixel and a cluster center. The difference is typically based on pixelcolor, intensity, texture, and location, or a weighted combination ofthese factors. K can be selected manually, randomly, or by a heuristic.This algorithm is guaranteed to converge, but it may not return theoptimal solution. The quality of the solution depends on the initial setof clusters and the value of K. While the k-means algorithm iseffective, it is no panacea for large databases like those of textdocuments and customer market data, which often include millions of datapoints. Applying the k-means algorithm in such cases can result inunacceptably long processing times and may exhaust the memory capacityof the processor implementing the algorithm. Moreover, although thek-means algorithm is guaranteed to find local optima, it is unlikely tofind global optima. Computational complexity is often described in termsof the number of floating point operations (flops) such as additions,multiplications, or comparisons required to perform the computation.

The global centroids can be arbitrary or set using k-means++ algorithm,or using other known algorithms. These can also be chosen as a functionof image attributes providing however that something is known about thecolor space of the input image. K-means++ is a probabilistic means ofinitialization for k-means clustering created by Arthur andVassilvitskii that has theoretical guarantees on expected outcomequality. Briefly, the k-means++ algorithm generates initial centers fork-means clustering. A first center c₁ is sampled from a uniformdistribution over a set of points X. Then, for k=2 to K: the k^(th)center c_(k) is sampled from a multi-nomial over X where point x hasprobability θ_(x) as defined by:

$\begin{matrix}{\theta_{x} = {\frac{{D(x)}^{2}}{\sum\limits_{x^{\prime} \in X}{D\left( x^{\prime} \right)}^{2}} \propto {D(x)}^{2}}} & (1)\end{matrix}$

where D(x) is the distance d to the closest center c_(k′), as given byD(x)=min_(k′<k) d(x,c _(k′))  (2)

In the present system, the work of computing the distance to theglobally known cluster centroids is allocated to a plurality ofdistributed code segments 1 through k. These code segments also computethe local cluster centroids—the centroids of the subset of pointsallocated to each distributed code segment. These two pieces of work arecarried out every iteration until convergence. The cluster centroids arecalculated and updated as the weighted centroids of the local centroids(i.e. for each segment), at the end of each iteration. Near synchronouscompletion of work that is assigned to the distributed code segments inevery iteration is essential; the lack of such synchronization mayresult in wasted time in this connected parallel computation. Thepresent method accounts for such synchronization by allocation of anequal quantum of work for each of the distributed code segments. Becauseof the connected parallel nature, each distributed code segment isrequired to complete the work assigned in about the same time as theothers for a given iteration. For this reason, the number of segments isselected as a function of the total number of resources available. Afterresource availability has been determined for a given job, the number ofsegments further depends on the speed requirements and the size of theinput files.

At step 410, upon completion of all the pixels, each resource computesthe centroid of its local clustering and the size of each cluster. Thepart centroids and cluster sizes are communicated for processing atapproximately the same time because of the nearly equal work that isapportioned to each resource.

At step 411, compute a weighted average of the part centroids to produceat a new global centroid. Here, the global centroid computation uses thepart sizes as weights.

Reference is now being made to FIG. 5 which is a continuation of theflow diagram of FIG. 4 with processing continuing with respect to nodeA.

At step 412, the centroids are checked for convergence. If not thenprocessing repeats, for a next iteration, with respect to node Bwherein, at step 412, the points to each global centroid are sorted by aminimum squared distance. The next iteration starts subsequently unlessthe centroids have converged. If, at step 420, convergence has beenachieved then image segmentation is complete. Upon computation of thesesegments, they are written to memory or storage. While writing these tostorage, the segments can be broken down into more than the number ofpixel processors. All become available simultaneously for the next stageof intra-image processing.

At step 414, read all the k^(th) part outputs of each of thesegmentation resources as the k^(th) image segment for post-segmentationpixel-level processing. Each of the pixel processors is preferablyoperating completely in parallel. As soon as a certain pixel processorcompletes, the pixel processors are assigned a new task. There isbenefit to the number of pixel processors being an integral multiple ofk. With the choice of k made previously, we ensure that all imagesegments can be processed by the pixel processors in parallel. By thesame token, if we want to perform the pixel processing in multiplestages, the image segments can be divided into an integral multiple of kso that at every sub-stage of pixel processing achieves a maximumparallelization. Because the pixel processors can apply differentprocessing techniques, semantics may be assigned to segment based uponsegment type in a pre-processing step. The segment type could denoteeither (a) a color-related type, or (b) an object-related type.Pre-processing of a segment prior to pixel processing helps to revealthe type of transformation that needs to be applied to that segment. Forexample, skin tones, facial features, trees, landscape, scenery etc, maybe processed differently. Any type of pixel processing can be introducedin this stage.

At step 416, perform pixel-level processing of the image segments.

At step 418, the pixels in each pixel-level processing block are sortedinto ordered batches. For increased bandwidth and image streaming,decide the number of pixels desired in each of the output batches, p.Parts of the image can be made available earlier or in a slightlydelayed fashion. This will produce (n div p)+1 total sorted batches ofpixels-per-pixel processor. Each of the batches may not comprise exactlyp pixels, but they are guaranteed to be sorted. The sorted batches helpwith the efficiency of merging. The fact that, in the pixel processingstage, the pixels are in order and can also be taken advantage of,although they may not have pixels for each of the batches.

At step 420, the ordered batches are hierarchically merged. This isanother stage that is difficult to parallelize in image processing.However, owing to the sorted lists created, merging can occur in lineartime. If we do this in parallel we achieve the merge in O(p log m) timewhere p is the number of pixels per merged batch and m is the number ofsorted lists that have to be merged (using known results in mergingsorted lists). Normally, m is small and p can be chosen such that:(n div p)+1≦min(w,d)  (3)

This will allow for fast merges by ensuring there are k resourcesavailable for the merge operation. Also, all the merges (except perhapsthe last one) may complete at nearly the same time achievingsynchronization. So we not only achieve higher speeds due toparallelization, but achieve synchronization as well. While the presentalgorithms have been designed to achieve synchronization, failures dooccur due to input and output latencies between the stages or otherconcurrent uses of the cluster. So, in order to better manage merging,so long as we know we have at least 2 sorted lists that need to bepresent in a particular merged batch, the merge process can be startedimmediately. As soon as the other lists arrive, they are merged with theresults of the previous merge. For example, if there are 5 sortedbatches to be merged to batch 1, there could be a merge sequence asfollows:merge(1,5)→merge({1,5},2), merge(3,4)→merge({1,2,5},{3,4})  (4)

At step 422, the merged batches are concatenated to produce a processedoutput image. Thereafter, further intra-image processing stops.

Reference is now briefly being made to FIG. 6 which depicts one exampleembodiment of a distributed (hierarchical) merge operation of n-sortedsegments where n=2. Sorted segments 601 and 602 each comprise aplurality of ordered portions 603 and 604, respectively. These aremerged into a plurality of merge batches, shown as 605A-D. The mergedbatches are concatenated to produce processed output image 606.

Average Time Complexity for a Uni-Processor System

For a uni-processor system, the average-case time complexity is given bythe following. Let n be the number of pixels, k the number of clusters,d the number of dimension and T the number of iterations. For k-means,the time complexity is: O(n*k*d*T). Pixel processing is linear in thenumber of pixels, O(n) and merging being n log n assuming merge sort.

Average Time Complexity for a Parallel System

Assuming there are going to k workers set according to availability andother considerations as above, clustering will take O(ndT) withsynchronization benefits. Profiling and grouping will take O(n/k) withsynchronization benefits. Merging will take O(n/k log k) again withsynchronization benefits. In total, theoretical estimates are of theorder of 1/k times the uni-processor version with synchronizationbenefits (owing to intertwining the choice of k with processoravailability w and desired number of processors d). In reality, therewill be I/O overheads as are dealing with large files resulting in lessthan k times speedup but with synchronization benefits.

In-Situ Pixel Processing in GPU

The algorithm hereof could be implemented on a GPU. The groups, parts,segments, etc., can be maintained as metadata associated with eachpixel. Metadata can be stored for a group of pixels of a given sectionor sub-section of the image in memory associated with the GPU along withthat of the (i,j)^(th) pixel. Pixel or group dependent processing may bestored in memory associated with that pixel. Optimizations include thefollowing considerations. 1) One can reduce the number of processingcores required based on pixel redundancy and GPU capacity in terms ofthe number of processing cores. 2) Several iterations of transfer backand forth to the GPU main memory may be required. The number oftrips/iterations can be pre-determined based on the number of CPU-basedcalculations that need to be performed. 3) Segment by segmentdifferentiated processing can be performed in multiple iterations. Inthis instance, each iteration can be given a certain pixel processingalgorithm to perform, as discussed above with respect to step 414.

Image Path Equivalences

In this section, we claim image path equivalence using the workflow onthe internal cloud. Methodology for quality improvements are not claimedherein. We conducted experiments using a high-resolution image tabloidcar image and the ADSB boundless image (1 thread: 9.43 minutes, 4threads: 7.8 minutes, 8 threads: 4.75 minutes).

This feasibility, in one embodiment, was performed on an iGen4 printengine with a FFPS DFE (Free Flow Print Server Digital Front End) and aninternal cloud. A 350 dpi image was used for the printing demonstration.The test image is in RGB color space and was processed using SBI(state-based imaging) image path on cloud processors. We carried outfollowing steps for the demonstration. (A) Process the RGB image with acustom global ICC profile. Create a CMYK color separated image byapplying the custom global ICC profile. All of this image processing isdone on the FFPS DFE and printed on iGen4. This is a normal print pathon iGen4 being used for printing any RGB images. We used a perceptualLUT from the destination profile to create the image. (B) Process theRGB image (tiff version) through the “state-based image path” on thecloud. We had created two segments. Both segments were processed usingthe perceptual LUT from the global destination profile. After mergingtwo segments using the approach described in FIG. 2, the state-basedimage path resulted in the creation of a CMYK image in tiff format. Weprinted this unified image through the native CMYK path on iGen4. Thisprocess completely removed the image processing in the FFPS DFE thatoccurred in step A. (C) Process the RGB image (tiff version) through the“state-based image path” on the cloud computer. We had created twosegments same as in step B. This time, we processed the segments asfollows. Segment one with global destination profile and segment twowith optimized destination profile for segment two. The segment twodestination profile was optimized to remove the artifacts present in theimage for the purpose of this demonstration. Both segments were thenmerged to obtain a CMYK image. We printed this image CMYK image throughthe native CMYK path on iGen4. This process again completely removed theimage processing in the FFPS DFE that occurred in step A. As expected,image quality artifacts present in the image were completely removedfrom the reproduced image.

In another embodiment, the pixel processing could be used forhandling/correcting blurry images, red-eye correction, and the like, invarious photo processing applications

Results demonstrated that printed images from steps A and B appearedvisually similar. These steps visually demonstrated the image pathequivalence (parallel processing being equivalent to uni-processorversions with increase in speed). Step C demonstrated the improvement toimage quality due to the use of optimized profile LUT local to the colorsegment. Thus the above architecture has been demonstrated to the extentthat we can indeed optimize the image for image quality local to thesegment/object based on the state of the image and the state of theprinter—on one printer for one critical image with two segments.

Example Special Purpose Computer System

Reference is now being made to FIG. 7 which illustrates a block diagramof one example embodiment of a special purpose computer system usefulfor implementing one or more aspects of the resources of FIGS. 1 and 2as further described with respect to the various flow diagrams of FIGS.4 and 5.

In FIG. 7, communications bus 702 serves as an information highwayinterconnecting the other illustrated components of special purposecomputer system 700. The special purpose computer incorporates a centralprocessing unit (CPU) 704 capable of executing machine readable programinstructions for performing any of the calculations, comparisons,logical operations, object identifications and extractions, user markingidentifications and extractions, and other program instructions forperforming the methods described above with respect to the flow diagramsand illustrated embodiments hereof. The CPU is in communication withRead Only Memory (ROM) 706 and Random Access Memory (RAM) 708 which,collectively, constitute main memory 707. Such memory may be used tostore machine readable program instructions and other program data andresults to sufficient to carry out any of the functionality describedherein.

Graphics Processing Unit (GPU) 705 performs pixel-level processing ofimage data and utilizes main memory 707 to store processed pixels in theform of batches of pixels.

Disk controller 710 interfaces with one or more storage devices 714.These storage devices may comprise external memory, zip drives, flashmemory, USB drives, memory sticks, or other storage devices withremovable media such as CD-ROM drive 712 and floppy drive 716. Machineexecutable program instructions for executing the methods hereof or forperforming any of the functionality shown with respect to theembodiments of FIGS. 1 and 2, for example. Any of the document imagesand bitmaps may also be stored on any of these storage devices. Examplecomputer readable media is, for example, a floppy disk, a hard-drive,memory, CD-ROM, DVD, tape, cassette, or other digital or analog media,or the like, which is capable of having embodied thereon a computerreadable program, one or more logical instructions, or other machineexecutable codes or commands that implement and facilitate the function,capability, and methodologies described herein. The computer readablemedium may additionally comprise computer readable information in atransitory state medium such as a network link and/or a networkinterface, including a wired network or a wireless network, which allowsthe computer system to read such computer readable information. Computerprograms (also called computer control logic) may be stored in a mainmemory and/or a secondary memory. Computer programs may also be receivedvia the communications interface. The computer readable medium isfurther capable of storing data, machine instructions, message packets,or other machine readable information, and may include non-volatilememory. Such computer programs, when executed, enable the computersystem to perform one or more aspects of the methods provided herein.Display interface 718 effectuates the display of information on displaydevice 720 in various formats such as, for instance, audio, graphic,text, and the like. Interface 724 effectuates a communication viakeyboard 726 and mouse 728. Communication with external devices mayoccur using example communication port(s) 722. Such ports may be placedin communication with any of the example networks shown and describedherein, such as the Internet or an intranet, either by direct (wired)link or wireless link. Example communication ports include modems,network cards such as an Ethernet card, routers, a PCMCIA slot and card,USB ports, and the like, capable of transferring data from one device toanother. Software and data transferred via any of the communicationports 722 are in the form of signals which may be any of digital,analog, electromagnetic, optical, infrared, or other signals capable ofbeing transmitted and/or received by the communications interface. Suchsignals may be implemented using, for example, a wire, cable, fiberoptic, phone line, cellular link, RF, or other signal transmission meanspresently known in the arts or which have been subsequently developed.

Various Embodiments

It will be appreciated that the above-disclosed and other features andfunctions, or alternatives thereof, may be desirably combined into manyother different systems or applications. Various presently unforeseen orunanticipated alternatives, modifications, variations, or improvementstherein may become apparent and/or subsequently made by those skilled inthe art which are also intended to be encompassed by the followingclaims. Accordingly, the embodiments set forth above are considered tobe illustrative and not limiting. Various changes to the above-describedembodiments may be made without departing from the spirit and scope ofthe invention. The teachings hereof can be implemented in hardware orsoftware using any known or later developed systems, structures,devices, and/or software by those skilled in the applicable art withoutundue experimentation from the functional description provided hereinwith a general knowledge of the relevant arts.

Moreover, the methods hereof can be implemented as a routine embedded ona personal computer or as a resource residing on a server orworkstation, such as a routine embedded in a plug-in, a photocopier, adriver, a scanner, a photographic system, a xerographic device, or thelike. The methods provided herein can also be implemented by physicalincorporation into an image processing or color management system.Furthermore, the teachings hereof may be partially or fully implementedin software using object or object-oriented software developmentenvironments that provide portable source code that can be used on avariety of computer, workstation, server, network, or other hardwareplatforms. One or more of the capabilities hereof can be emulated in avirtual environment as provided by an operating system, specializedprograms or leverage off-the-shelf computer graphics software such asthat in Windows, Java, or from a server or hardware accelerator or otherimage processing devices.

One or more aspects of the methods described herein are intended to beincorporated in an article of manufacture, including one or morecomputer program products, having computer usable or machine readablemedia. The article of manufacture may be included on at least onestorage device readable by a machine architecture or other xerographicor image processing system embodying executable program instructionscapable of performing the methodology described herein. The article ofmanufacture may be included as part of a xerographic system, anoperating system, a plug-in, or may be shipped, sold, leased, orotherwise provided separately either alone or as part of an add-on,update, upgrade, or product suite.

It will be appreciated that various of the above-disclosed and otherfeatures and functions, or alternatives thereof, may be combined intoother systems or applications. Various presently unforeseen orunanticipated alternatives, modifications, variations, or improvementstherein may become apparent and/or subsequently made by those skilled inthe art which are also intended to be encompassed by the followingclaims. Accordingly, the embodiments set forth above are considered tobe illustrative and not limiting.

Various changes to the above-described embodiments may be made withoutdeparting from the spirit and scope of the invention. The teachings ofany printed publications including patents and patent applications, areeach separately hereby incorporated by reference in their entirety.

What is claimed is:
 1. A method for parallel processing of intra-imagedata in a state-based imaging system, the method comprising: receivingat least one customer expectation criteria with respect to a customerjob intended to be processed by a state-based imaging system comprisingw available resources, said customer criteria comprising at least a typeof processing desired to be performed on said customer's job; selectinga desired number d of said available resources in said state-basedimaging system which at least meet said customer's expectation criteria;using at least one segmentation resource to segment said image intok=min(w,d) image segments; using k pixel-level processors to processsaid k image segments into batches of processed pixels, each pixel-levelprocessor operating on a respective k^(th) segment in parallel; using atleast one sorting resource to sort said batches of processed pixels intoa plurality of ordered batches; using at least one merging resource tohierarchically merge said ordered batches into a plurality of mergedbatches of processed pixels; and concatenating said merged batches ofprocessed pixels to obtain a processed image.
 2. The method of claim 1,wherein said state-based imaging system comprises any of: a plurality ofprocessors, a plurality of multi-core processors, and a multi-threadedsystem.
 3. The method of claim 1, wherein any of said w availableresources comprises a distributed code segment.
 4. The method of claim1, wherein segmenting of said image is performed using a distributedsegmentation algorithm that selects said k number of image segments. 5.The method of claim 1, wherein said selection of said desired number ofavailable resources is based upon any of: a prior run history of similarjobs, and a dynamic optimization strategy.
 6. The method of claim 1,wherein said customer expectation criteria comprises any of: a desiredprice point for said customer job; a desired timeframe in which saidcustomer job is to be processed; and a desired quality of said processedoutput image.
 7. The method of claim 1, further comprising billing saidcustomer for said job according to a billing criteria comprising any of:said desired number of resources d used to process said customer job;said number of pixels in said received image; said total number ofpixels processed by each of said pixel-level processing resources; saidtotal number of sorted batches produced by said sorting resources; saidtotal number of batches merge by said merging resources; and a size ofsaid processed output image.
 8. The method of claim 1, wherein saidselection of said desired number of available resources is based upon aprior run history of similar jobs.
 9. A state-based imaging system forparallel processing of intra-image data, said state-based imaging systemcomprising: at least one segmentation resource for segmenting an inputimage into k=min(w,d) image segments, where w is a number of availableresources in said stage-based imaging system and d is a number of saidavailable resources selected to at least meet a customer's expectationcriteria comprising at least a type of processing desired to beperformed on said customer's job; a plurality of pixel-level processorsfor processing said k image segments into batches of processed pixels,each processor operating on a respective k^(th) segment in parallel; atleast one sorting resource for sorting said batches of processed pixelsinto ordered batches of processed pixels; at least one merging resourcefor hierarchically merging said ordered batches to produce a pluralityof merged batches of processed pixels; and a processor executing machinereadable instructions for concatenating said plurality of merged batchesof processed pixels to obtain a processed image.
 10. The system of claim9, wherein said state-based imaging system comprises any of: a pluralityof processors, a plurality of multi-core processors, and amulti-threaded system.
 11. The system of claim 9, wherein any of said wavailable resources comprises a distributed code segment.
 12. The systemof claim 9, wherein segmenting of said image is performed using adistributed segmentation algorithm that selects said k number of imagesegments.
 13. The system of claim 9, wherein said selection of saiddesired number of available resources is based upon any of: a prior runhistory of similar jobs, and a dynamic optimization strategy.
 14. Thesystem of claim 9, wherein said customer expectation criteria comprisesany of: a desired price point for said customer job; a desired timeframein which said customer job is to be processed; and a desired quality ofsaid processed output image.
 15. The system of claim 9, furthercomprising billing said customer for said job according to a billingcriteria comprising any of: said desired number of resources d used toprocess said customer job; said number of pixels in said received image;said total number of pixels processed by each of said pixel-levelprocessing resources; said total number of sorted batches produced bysaid sorting resources; said total number of batches merge by saidmerging resources; and a size of said processed output image.
 16. Thesystem of claim 9, wherein said selection of said desired number ofavailable resources is based upon a prior run history of similar jobs.